Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: Sinkevich Date: Oct 5, 2013 5:59am
Forum: web Subject: Please, check my robots.txt

Good day.
I have a forum.

I want that ia_archiver crawls the index page and pages of partitions of forum.

myforum.com
myforum.com/forum1.html
myforum.com/forum2.html
myforum.com/forum3.html
...

But I want that ia_archiver will delete all pages of topics of forum from archive.org base.

myforum.com/topic1.html
myforum.com/topic2.html
myforum.com/topic3.html
...
myforum.com/topic99999.html

Please, check my robots.txt.
Is it true?

User-agent: ia_archiver
Disallow: /topic